PowerAI DDL
نویسندگان
چکیده
As deep neural networks become more complex and input data-sets grow larger, it can take days or even weeks to train a deep neural network to the desired accuracy. Therefore, distributed Deep Learning at a massive scale is a critical capability, since it offers the potential to reduce the training time from weeks to hours. In this paper, we present a software-hardware co-optimized distributed Deep Learning system that can achieve near-linear scaling up to hundreds of GPUs. The core algorithm is a multi-ring communication pattern that provides a good tradeoff between latency and bandwidth and adapts to a variety of system configurations. The communication algorithm is implemented as a library for easy use. This library has been integrated into Tensorflow, Caffe, and Torch. We train Resnet-101 on Imagenet 22K with 64 IBM Power8 S822LC servers (256 GPUs) in about 7 hours to an accuracy of 33.8% validation accuracy. Microsoft’s ADAM [10] and Google’s DistBelief [11] results did not reach 30% validation accuracy for Imagenet 22K. Compared to Facebook’s recent paper [1] on 256 GPU training, we use a different communication algorithm, and our combined software and hardware system offers better communication overhead for Resnet50. A PowerAI DDL enabled version of Torch completed 90 epochs of training on Resnet 50 for 1K classes in 50 minutes using 64 IBM Power8 S822LC servers (256 GPUs).
منابع مشابه
The Influence of Data-Driven Exercises Through Using a Computer Program on Vocabulary Improvement in an EFL Context
The present study was conducted to evaluate data driven learning (DDL) combined with Computer Assisted Language Learning (CALL) as an approach to improving vocabulary knowledge of Iranian postgraduates majoring in teaching English, English literature and translation. The purpose was to help language learners get familiar with DDL as a student-centered method taking advantage of a computer progr...
متن کاملDike: Revisiting Resource Management for Distributed Deep Learning
The recent adoption of deep learning for diverse applications has required scaling infrastructures both horizontally and vertically. As a result, efficient resource management for distributed deep learning (DDL) frameworks is becoming increasingly important. However, existing techniques for scaling DDL applications rely on general-purpose resource managers originally designed for data intensive...
متن کاملHSF-1 Regulators DDL-1/2 Link Insulin-like Signaling to Heat-Shock Responses and Modulation of Longevity
Extended longevity is often correlated with increased resistance against various stressors. Insulin/IGF-1-like signaling (IIS) is known to have a conserved role in aging and cellular mechanisms against stress. In C. elegans, genetic studies suggest that heat-shock transcription factor HSF-1 is required for IIS to modulate longevity. Here, we report that the activity of HSF-1 is regulated by IIS...
متن کاملEssence of Re-Calibrating Optical Instruments: Analysis of the Digital Delay Line
In this work a digital delay line (DDL) from Oz Optics was used. The DDL introduces a delay in optical paths between the fast and slow orthogonal polarizations by adjusting the relative travel distance of the two orthogonal modes. The insertion loss of the DDL is 1.5 dB. A Labview programme was used to write and read from the DDL. The DDL was adjusted from 0 to 60 ps and the observed first orde...
متن کامل.ndomized Study of Didanosine Monotherapy and Combination Therapy With Zidovudine in Hemophilic and Nonhemophilic Subjects With Asymptomatic Human Immunodeficiency Virus-l Infection
To evaluate the safety and efficacy of didanosine (ddl) monotherapy and three different combinations of zidovudine (ZDV) and ddl in asymptomatic human immunodeficiency virus-l (HIV-1) infection, we conducted an open-label, phase 1/11 study in 126 asymptomatic HIV-l-infected hemophilic and nonhemophilic subjects with a CD4 count of 200 to 500/mm3 stratified for prior zidovudine treatment and bas...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید
ثبت ناماگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید
ورودعنوان ژورنال:
- CoRR
دوره abs/1708.02188 شماره
صفحات -
تاریخ انتشار 2017